The present invention relates generally to distributed systems, and more particularly, to dynamic borderline tracing for tracking message flows across distributed systems.
The following references [1]-[9] are noted herein:    [1]. Tuxedo http://docs.oracle.com/cd/E15261_01/tuxedo/docs11gr1/int/intatm.html    [2]. LD_PRELOAD http://www.linuxjournal.com/article/7795.    [3]. Galen Hunt and Doug Brubacher, Detours: Binary Interception of Win32 Functions, in Third USENIX Windows NT Symposium, USENIX, July 1999.    [4]. C.-K. Luk, R. Cohn, R. Muth, H. Patil, A. Klauser, G. Lowney, S. Wallace, V. J. Reddi, and K. Hazelwood. Pin: building customized program analysis tools with dynamic instrumentation. In Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, PLDI '05, pages 190-200, New York, N.Y., USA, 2005. ACM.    [5]. B. M. Cantrill, M. W. Shapiro, and A. H. Leventhal. Dynamic instrumentation of production systems. In Proceedings of the annual conference on USENIX Annual Technical Conference, ATEC '04, pages 2-2, Berkeley, Calif., USA, 2004. USENIX Association.    [6]. Eric Koskinen and John Jannotti. 2008. BorderPatrol: isolating events for black-box tracing. SIGOPS Oper. Syst. Rev. 42, 4 (April 2008), 191-203. DOI=10.1145/1357010.1352613 http://doi.acm.org/10.1145/1357010.1352613.    [7]. Bryan Buck and Jeffrey K. Hollingsworth. 2000. An API for Runtime Code Patching. Int. J. High Perform. Comput. Appl. 14, 4 (November 2000), 317-329. DOI=10.1177/109434200001400404 http://dx.doi.org/10.1177/109434200001400404    [8]. SystemTap http://sourceware.org/systemtap/.    [9]. Arora, N.; Hui Zhang; Junghwan Rhee; Yoshihira, K.; Guofei Jiang, “iProbe: A lightweight user-level dynamic instrumentation tool,” Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Conference on, vol., no., pp. 742,745, 11-15 Nov. 2013.
Obtaining precise request traces in distributed systems can be a difficult task. Traditional log data provided by applications, using system call logs is usually insufficient to generate end-to end transaction flow. Information in trace logs from applications either contains user-request data or occasionally debug data, which is too sparse to generate the flow. On the other hand, system calls are too low-layered and are often not useful to the developers in debugging any problems (since user-space information is usually more valuable).
In distributed applications the request spans across several modules and the execution flow is multiplexed across modules in nodes across the distributed system. These modules are usually written by different teams and provide a clear layer of communication (API Function) between them. Capturing these API functions provides optimal granularity to show inter-module communication while not incurring a high overhead. In this invention we aim to utilize this layer of communication to generate transaction flow across distributed systems. Additionally, we present an analysis of the message logs generate from our approach on message queuing systems (e.g. Tuxedo[1]), to show end-to-end transaction flow tracking.
Dynamic instrumentation is a well studied field and there are several tools, [4], [5], that provide such instrumentation capability. These tools usually have high overhead and may or may not require access to the source code. Since we need a black-box mechanism which is lightweight, we use iProbe here as the basis of our dynamic instrumentation as it allows us to turn on and off instrumentation inside without any modification of the source code.
The closest work towards dynamic border-line tracing is BorderPatrol [6]. See Eric Koskinen and John Jannotti. 2008. BorderPatrol: isolating events for black-box tracing. SIGOPS Oper. Syst. Rev. 42, 4 (April 2008), 191203.DOI=10.1145/1357010.1352613, http://doi.acm.org/10.1145/1357010.1352613.
The BorderPatrol explores tracing API libraries. These API calls can then be used as breadcrumbs to generate the transaction flow. However, unlike our approach, BorderPatrol focuses on system library API calls, also it does not provide any dynamic instrumentation facility. Other dynamic instrumentation tools include PIN[4], dyninst[7], SystemTap[8], DTrace[5] etc. However, these tools have an extremely high overhead or are unable to capture user-space functions.
Accordingly, there is a need for a dynamic borderline tracing for tracking message flows across distributed systems.